﻿using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

using lucere.toolkit;

namespace lucere.analysis
{
    /// <summary> A <code>TokenStream</code> enumerates the sequence of tokens, either from
    /// {@link Field}s of a {@link Document} or from query text.
    /// <p/>
    /// This is an abstract class. Concrete subclasses are:
    /// <ul>
    /// <li>{@link Tokenizer}, a <code>TokenStream</code> whose input is a Reader; and</li>
    /// <li>{@link TokenFilter}, a <code>TokenStream</code> whose input is another
    /// <code>TokenStream</code>.</li>
    /// </ul>
    /// A new <code>TokenStream</code> API has been introduced with Lucene 2.9. This API
    /// has moved from being {@link Token} based to {@link Attribute} based. While
    /// {@link Token} still exists in 2.9 as a convenience class, the preferred way
    /// to store the information of a {@link Token} is to use {@link AttributeImpl}s.
    /// <p/>
    /// <code>TokenStream</code> now extends {@link AttributeSource}, which provides
    /// access to all of the token {@link Attribute}s for the <code>TokenStream</code>.
    /// Note that only one instance per {@link AttributeImpl} is created and reused
    /// for every token. This approach reduces object creation and allows local
    /// caching of references to the {@link AttributeImpl}s. See
    /// {@link #IncrementToken()} for further details.
    /// <p/>
    /// <b>The workflow of the new <code>TokenStream</code> API is as follows:</b>
    /// <ol>
    /// <li>Instantiation of <code>TokenStream</code>/{@link TokenFilter}s which add/get
    /// attributes to/from the {@link AttributeSource}.</li>
    /// <li>The consumer calls {@link TokenStream#Reset()}.</li>
    /// <li>The consumer retrieves attributes from the stream and stores local
    /// references to all attributes it wants to access</li>
    /// <li>The consumer calls {@link #IncrementToken()} until it returns false and
    /// consumes the attributes after each call.</li>
    /// <li>The consumer calls {@link #End()} so that any end-of-stream operations
    /// can be performed.</li>
    /// <li>The consumer calls {@link #Close()} to release any resource when finished
    /// using the <code>TokenStream</code></li>
    /// </ol>
    /// To make sure that filters and consumers know which attributes are available,
    /// the attributes must be added during instantiation. Filters and consumers are
    /// not required to check for availability of attributes in
    /// {@link #IncrementToken()}.
    /// <p/>
    /// You can find some example code for the new API in the analysis package level
    /// Javadoc.
    /// <p/>
    /// Sometimes it is desirable to capture a current state of a <code>TokenStream</code>
    /// , e. g. for buffering purposes (see {@link CachingTokenFilter},
    /// {@link TeeSinkTokenFilter}). For this usecase
    /// {@link AttributeSource#CaptureState} and {@link AttributeSource#RestoreState}
    /// can be used.
    /// </summary>
    public interface ITokenStream : IAttributeSource, IDisposable
    {
        /// <summary> Consumers (i.e., {@link IndexWriter}) use this method to advance the stream to
        /// the next token. Implementing classes must implement this method and update
        /// the appropriate {@link AttributeImpl}s with the attributes of the next
        /// token.
        /// 
        /// The producer must make no assumptions about the attributes after the
        /// method has been returned: the caller may arbitrarily change it. If the
        /// producer needs to preserve the state for subsequent calls, it can use
        /// {@link #captureState} to create a copy of the current attribute state.
        /// 
        /// This method is called for every token of a document, so an efficient
        /// implementation is crucial for good performance. To avoid calls to
        /// {@link #AddAttribute(Class)} and {@link #GetAttribute(Class)} or downcasts,
        /// references to all {@link AttributeImpl}s that this stream uses should be
        /// retrieved during instantiation.
        /// 
        /// To ensure that filters and consumers know which attributes are available,
        /// the attributes must be added during instantiation. Filters and consumers
        /// are not required to check for availability of attributes in
        /// {@link #IncrementToken()}.
        /// 
        /// </summary>
        /// <returns> false for end of stream; true otherwise
        /// </returns>
        bool IncrementToken();

        /// <summary> This method is called by the consumer after the last token has been
        /// consumed, after {@link #IncrementToken()} returned <code>false</code>
        /// (using the new <code>TokenStream</code> API). Streams implementing the old API
        /// should upgrade to use this feature.
        /// <p/>
        /// This method can be used to perform any end-of-stream operations, such as
        /// setting the final offset of a stream. The final offset of a stream might
        /// differ from the offset of the last token eg in case one or more whitespaces
        /// followed after the last token, but a {@link WhitespaceTokenizer} was used.
        /// 
        /// </summary>
        /// <throws>  IOException </throws>
        void End();

        /// <summary> Resets this stream to the beginning. This is an optional operation, so
        /// subclasses may or may not implement this method. {@link #Reset()} is not needed for
        /// the standard indexing process. However, if the tokens of a
        /// <code>TokenStream</code> are intended to be consumed more than once, it is
        /// necessary to implement {@link #Reset()}. Note that if your TokenStream
        /// caches tokens and feeds them back again after a reset, it is imperative
        /// that you clone the tokens when you store them away (on the first pass) as
        /// well as when you return them (on future passes after {@link #Reset()}).
        /// </summary>
        void Reset();
    }
}
